Note

This page corresponds to a Jupyter notebook you can try out yourself.
(The original version is here.)

---

Query Tutorial

# The plotting examples below require holoviews, hvplot, and bokeh:
# conda install -c conda-forge bokeh holoviews hvplot
import numpy as np
import pandas as pd

import bokeh
import hvplot.pandas
import holoviews as hv

import bokeh.palettes
from bokeh.plotting import figure, show, output_notebook
output_notebook()

Loading BokehJS ...

Create a Client

Before you begin, you must create a Client object. It will be stored globally and used for all communication with the neuprint server.

Initialize it with your personal authentication token. See the Quickstart guide for details.

from neuprint import Client

TOKEN = "" # <--- Paste your token here
           # (or define NEUPRINT_APPLICATION CREDENTIALS in your environment)

c = Client('neuprint.janelia.org', 'hemibrain:v1.0.1', TOKEN)

ROIs

In neuprint, each neuron is annotated with the list of regions (ROIs) it intersects, along with the synapse counts in each.

The ROIs comprise a hierarchy, with smaller ROIs nested within larger ROIs. Furthermore, primary ROIs are guaranteed not to overlap, and they roughly tile the entire brain (with some gaps).

For a quick overview of the ROI hierarchy, use fetch_roi_hierarchy().

from neuprint import fetch_roi_hierarchy

# Show the ROI hierarchy, with primary ROIs marked with '*'
print(fetch_roi_hierarchy(False, mark_primary=True, format='text'))

hemibrain
 +-- AL(L)*
 +-- AL(R)*
 +-- AOT(R)
 +-- CX
 |   +-- AB(L)*
 |   +-- AB(R)*
 |   +-- EB*
 |   +-- FB*
 |   +-- NO*
 |   +-- PB*
 +-- GC
 +-- GF(R)
 +-- GNG*
 +-- INP
 |   +-- ATL(L)*
 |   +-- ATL(R)*
 |   +-- CRE(L)*
 |   +-- CRE(R)*
 |   +-- IB*
 |   +-- ICL(L)*
 |   +-- ICL(R)*
 |   +-- SCL(L)*
 |   +-- SCL(R)*
 +-- LH(R)*
 +-- LX(L)
 |   +-- BU(L)*
 |   +-- LAL(L)*
 +-- LX(R)
 |   +-- BU(R)*
 |   +-- LAL(R)*
 +-- MB(+ACA)(R)
 |   +-- MB(R)
 |   |   +-- CA(R)*
 |   |   +-- PED(R)*
 |   |   +-- a'L(R)*
 |   |   +-- aL(R)*
 |   |   +-- b'L(R)*
 |   |   +-- bL(R)*
 |   |   +-- gL(R)*
 |   +-- dACA(R)
 |   +-- lACA(R)
 |   +-- vACA(R)
 +-- MB(L)
 |   +-- CA(L)*
 |   +-- a'L(L)*
 |   +-- aL(L)*
 |   +-- b'L(L)*
 |   +-- bL(L)*
 |   +-- gL(L)*
 +-- OL(R)
 |   +-- AME(R)*
 |   +-- LO(R)*
 |   +-- LOP(R)*
 |   +-- ME(R)*
 +-- PENP
 |   +-- CAN(R)*
 |   +-- FLA(R)*
 |   +-- PRW*
 |   +-- SAD*
 +-- POC
 +-- SNP(L)
 |   +-- SIP(L)*
 |   +-- SMP(L)*
 +-- SNP(R)
 |   +-- SIP(R)*
 |   +-- SLP(R)*
 |   +-- SMP(R)*
 +-- VLNP(R)
 |   +-- AOTU(R)*
 |   +-- AVLP(R)*
 |   +-- PLP(R)*
 |   +-- PVLP(R)*
 |   +-- WED(R)*
 +-- VMNP
 |   +-- EPA(L)*
 |   +-- EPA(R)*
 |   +-- GOR(L)*
 |   +-- GOR(R)*
 |   +-- IPS(R)*
 |   +-- SPS(L)*
 |   +-- SPS(R)*
 |   +-- VES(L)*
 |   +-- VES(R)*
 +-- mALT(L)
 +-- mALT(R)

Neuron Search Criteria

Specify neurons of interest by bodyId, type/instance, or via a NeuronCriteria object. With NeuronCriteria, you can specify multiple search constraints, including the ROIs in which matched neurons must contain synapses.

from neuprint import NeuronCriteria as NC

# Example: Select a specific body
criteria = 387023620
criteria = NC(bodyId=387023620)

# Example: Select several bodies
criteria = [387023620, 387364605, 416642425]
criteria = NC(bodyId=[387023620, 387364605, 416642425])

# Example: Select bodies by exact type
criteria = 'PEN_b(PEN2)'
criteria = NC(type='PENPEN_b(PEN2)')

# Example: Select bodies by exact instance
criteria = 'PEN(PB06)_b_L4'
criteria = NC(type='PEN(PB06)_b_L4')

# Example: Select bodies by type name pattern
criteria = NC(type='PEN.*', regex=True)

# Example: Select bodies by region (input or output)
criteria = NC(rois=['PB', 'EB'])

# Example: Select traced neurons which intersect the PB ROI with at least 100 inputs (PSDs).
criteria = NC(inputRois=['PB'], min_roi_inputs=100, status='Traced', cropped=False)

Fetch neuron properties

Neuron properties and per-ROI synapse distributions can be obtained with fetch_neurons(). Two dataframes are returned: one for neuron properties, and one for the counts of synapses in each ROI.

from neuprint import fetch_neurons
neuron_df, roi_counts_df = fetch_neurons(criteria)

The total count of pre-synaptic and post-synaptic points within each neuron are given in the pre and post columns:

neuron_df[['bodyId', 'instance', 'type', 'pre', 'post', 'status', 'cropped', 'size']]

	bodyId	instance	type	pre	post	status	cropped	size
0	387023620	PEN(PB06)_b_L4	PEN_b(PEN2)	343	1714	Traced	False	685261005
1	387364605	EPG(PB08)_L3	EPG	645	4333	Traced	False	1459837829
2	416642425	EPG(PB08)_R4	EPG	470	3828	Traced	False	1165123520
3	417998204	PFNv(PB05)_R4	PFNv	229	990	Traced	False	473864774
4	419060083	PFGs(PB10)_L2	PFGs	556	2395	Traced	False	1103881970
...	...	...	...	...	...	...	...	...
476	5813080979	PEN(PB06)_a_L5	PEN_a(PEN1)	544	1595	Traced	False	924457147
477	5813081494	PFR_b(PB11b)_R1	PFR_b	118	1891	Traced	False	588428634
478	5813081686	PFNa(PB03)_L8	PFNa	121	510	Traced	False	283961924
479	5813087532	IbSpsP(PB17)_R9	IbSpsP	108	388	Traced	False	257354274
480	5813128308	PFNp(PB01)_a_L7	PFNp_a	71	310	Traced	False	187208043

481 rows × 8 columns

The per-ROI synapse counts are returned in the second DataFrame.

Note: Since ROIs overlap (see hierarchy above), the sum of the per-ROI counts for each body will be more than the pre and post columns above.

roi_counts_df

	bodyId	roi	pre	post
0	387023620	CX	341	1704
1	387023620	PB	1	777
2	387023620	PB(L5)	0	33
3	387023620	PB(L4)	1	723
4	387023620	INP	0	5
...	...	...	...	...
7289	5813128308	FB	42	33
7290	5813128308	FBl1	42	33
7291	5813128308	NO	27	107
7292	5813128308	NO(R)	27	107
7293	5813128308	NO3(R)	27	107

7294 rows × 4 columns

Fetch connections

Find synaptic connection strengths between one set of neurons and another using fetch_adjacencies().

The “source” and/or “target” neurons are selected using NeuronCriteria. Additional parameters allow you to filter by connection strength or ROI. Two DataFrames are returned, for neuron properties and per-ROI connection strengths.

from neuprint import fetch_adjacencies, NeuronCriteria as NC

# Example: Fetch all downstream connections FROM a set of neurons
neuron_df, conn_df = fetch_adjacencies([387023620, 387364605, 416642425], None)

# Example: Fetch all upstream connections TO a set of neurons
neuron_df, conn_df = fetch_adjacencies(None, [387023620, 387364605, 416642425])

# Example: Fetch all direct connections between a set of upstream neurons and downstream neurons
neuron_df, conn_df = fetch_adjacencies(NC(type='Delta.*', regex=True), NC(type='PEN.*', regex=True))

conn_df.sort_values('weight', ascending=False)

	bodyId_pre	bodyId_post	roi	weight
179	910783961	5813070465	PB	184
57	880880259	849421763	PB	141
316	911565419	5813070465	PB	141
92	910442723	849421763	PB	139
216	911129204	724280817	PB	127
...	...	...	...	...
351	911578496	724280817	PB	1
349	911578496	664645558	PB	1
348	911578496	634608104	ATL(L)	1
346	911578496	509410587	PB	1
660	5813061383	5813070465	PB	1

657 rows × 4 columns

Merge arbitrary neuron properties onto the connection table with merge_neuron_properties()

from neuprint import merge_neuron_properties

conn_df = merge_neuron_properties(neuron_df, conn_df, ['type', 'instance'])
conn_df

	bodyId_pre	bodyId_post	roi	weight	type_pre	instance_pre	type_post	instance_post
0	734581598	387023620	PB	1	Delta7_a	Delta7(PB15)_a_L	PEN_b(PEN2)	PEN(PB06)_b_L4
1	734581598	508793049	PB	1	Delta7_a	Delta7(PB15)_a_L	PEN_a(PEN1)	PEN(PB06)_a_L7
2	734581598	509410587	PB	3	Delta7_a	Delta7(PB15)_a_L	PEN_a(PEN1)	PEN(PB06)_a_R4
3	734581598	569775058	PB	17	Delta7_a	Delta7(PB15)_a_L	PEN_b(PEN2)	PEN(PB06)_b_R4
4	734581598	570461892	PB	21	Delta7_a	Delta7(PB15)_a_L	PEN_a(PEN1)	PEN(PB06)_a_R3
...	...	...	...	...	...	...	...	...
652	5813061383	1508334312	PB	1	Delta7_b	Delta7(PB15)_b_R	PEN_a(PEN1)	PEN(PB06)_a_L3
653	5813061383	1600083889	PB	37	Delta7_b	Delta7(PB15)_b_R	PEN_b(PEN2)	PEN(PB06)_b_L9
654	5813061383	1601771527	PB	4	Delta7_b	Delta7(PB15)_b_R	PEN_b(PEN2)	PEN(PB06)_b_R9
655	5813061383	5813047157	PB	12	Delta7_b	Delta7(PB15)_b_R	PEN_b(PEN2)	PEN(PB06)_b_L2
656	5813061383	5813070465	PB	1	Delta7_b	Delta7(PB15)_b_R	PEN_b(PEN2)	PEN(PB06)_b_L3

657 rows × 8 columns

Connection Matrix

You can convert a connection table into a connectivity matrix via connection_table_to_matrix().

from neuprint.utils import connection_table_to_matrix

matrix = connection_table_to_matrix(conn_df, 'bodyId', sort_by='type')
matrix.iloc[:10, :10]

bodyId_post	634608104	1508334312	664645558	509410587	5813056953	508793049	1125964630	910447075	849482511	847336755
bodyId_pre
850182049	0	0	0	0	0	0	0	0	0	0
1067331159	0	0	0	0	0	0	0	0	0	0
1129396529	0	0	0	0	0	0	0	0	0	0
755950242	0	0	0	0	0	0	0	0	0	0
1225601395	0	0	0	1	1	1	0	0	0	0
1009043211	0	0	0	0	0	0	0	0	0	0
946300031	0	0	0	0	0	0	0	0	0	0
911906936	20	13	0	0	0	0	0	0	0	15
911901802	0	4	0	0	1	0	0	0	33	17
911911699	37	16	0	0	0	0	0	0	1	0

matrix.hvplot.heatmap(height=600, width=700).opts(xrotation=60)

Data type cannot be displayed:

matrix = connection_table_to_matrix(conn_df, ('bodyId_pre', 'type_post'))
matrix.hvplot.heatmap(height=600, width=400)

Data type cannot be displayed:

Synapses

Fetch synapses for a set of bodies using NeuronCriteria, and optionally apply additional filtering with SynapseCriteria.

from neuprint import fetch_synapses, NeuronCriteria as NC, SynapseCriteria as SC

neuron_criteria = NC(status='Traced', type='FB4Y', cropped=False, inputRois=['EB'], min_roi_inputs=100, min_pre=400)
eb_tbar_criteria = SC(rois='EB', type='pre', primary_only=True)
eb_tbars = fetch_synapses(neuron_criteria, eb_tbar_criteria)

# Plot the synapse positions in a 2D projection
p = figure()
p.scatter(eb_tbars['x'], eb_tbars['z'])
p.y_range.flipped = True
show(p)

Synapse Connections

Here, we fetch all synapse-synapse connections from a set of neurons. Provide a NeuronCriteria for the source or target neurons (or both) to filter the neurons of interest, and optionally filter the synapses themselves via SynapseCriteria.

from neuprint import fetch_synapse_connections
eb_syn_criteria = SC(rois='EB', primary_only=True)
eb_conns = fetch_synapse_connections(neuron_criteria, None, eb_syn_criteria)
eb_conns.head()

	bodyId_pre	bodyId_post	roi_pre	roi_post	x_pre	y_pre	z_pre	x_post	y_post	z_post	confidence_pre	confidence_post
0	5813027016	978733459	EB	EB	26199	23565	18002	26210	23553	18003	0.977	0.947756
1	5813027016	978733459	EB	EB	25559	25038	23046	25554	25038	23056	0.928	0.920682
2	5813027016	978733459	EB	EB	23472	24677	21980	23464	24674	21970	0.683	0.692000
3	5813027016	978733459	EB	EB	28490	25331	22543	28519	25332	22545	0.956	0.752377
4	5813027016	978733459	EB	EB	29141	24584	19235	29129	24559	19240	0.953	0.334432

Let’s determine the post-synaptic neuron types, and plot the synapses for the top 5 types.

# Retrieve the types of the post-synaptic neurons
post_neurons, _ = fetch_neurons(eb_conns['bodyId_post'].unique())
eb_conns = merge_neuron_properties(post_neurons, eb_conns, 'type')

top5_counts = eb_conns['type_post'].value_counts().head(5)
top5_counts

EPG            1480
EQ5             571
PEN_a(PEN1)     374
ExR7            264
ExR4            230
Name: type_post, dtype: int64

colormap = dict(zip(top5_counts.index, bokeh.palettes.Category10[5]))
points = eb_conns.query('type_post in @top5_counts.index').copy()
points['color'] = points['type_post'].map(colormap)

p = figure()
p.scatter(points['x_post'], points['z_post'], color=points['color'])
p.y_range.flipped = True
show(p)

Skeletons

Download skeletons with Client.fetch_skeleton().

# Download some skeletons as DataFrames and attach columns for bodyId and color
skeletons = []
for i, bodyId in enumerate(eb_conns['bodyId_pre'].unique()):
    s = c.fetch_skeleton(bodyId, format='pandas')
    s['bodyId'] = bodyId
    s['color'] = bokeh.palettes.Accent[5][i]
    skeletons.append(s)

# Combine into one big table for convenient processing
skeletons = pd.concat(skeletons, ignore_index=True)
skeletons.head()

	rowId	x	y	z	radius	link	bodyId	color
0	1	21626.0	34402.0	12790.0	10.0000	-1	5813027016	#7fc97f
1	2	21626.0	34402.0	12810.0	10.0000	1	5813027016	#7fc97f
2	3	21626.0	34382.0	12830.0	10.0000	2	5813027016	#7fc97f
3	4	21626.0	34362.0	12850.0	18.2843	3	5813027016	#7fc97f
4	5	21606.0	34342.0	12870.0	18.2843	4	5813027016	#7fc97f

# Join parent/child nodes for plotting as line segments below.
segments = skeletons.merge(skeletons, 'inner',
                           left_on=['bodyId', 'rowId'],
                           right_on=['bodyId', 'link'],
                           suffixes=['_child', '_parent'])

p = figure()
p.y_range.flipped = True

# Plot skeleton segments (in 2D)
p.segment(x0='x_child', x1='x_parent',
          y0='z_child', y1='z_parent',
          color='color_child',
          source=segments)

# Also plot the synapses from the above example
p.scatter(points['x_post'], points['z_post'], color=points['color'])

show(p)

Cypher Logging

Tip: Inspect all cypher queries by enabling debug logging.

from neuprint.client import setup_debug_logging
setup_debug_logging()

synapses = fetch_synapses(5813027016, SC(rois='EB', type='post'))

[2020-06-30 05:18:46,776] DEBUG Performing cypher query against dataset 'hemibrain:v1.0.1':


    MATCH (n:Neuron)
    WHERE
      // -- Basic conditions for segment 'n' --
      n.bodyId = 5813027016
    RETURN n.bodyId as bodyId

[2020-06-30 05:18:46,811] DEBUG Performing cypher query against dataset 'hemibrain:v1.0.1':

    MATCH (n:Neuron)
    WHERE
      // -- Basic conditions for segment 'n' --
      n.bodyId = 5813027016
      AND (n.`EB`)

    MATCH (n)-[:Contains]->(ss:SynapseSet),
          (ss)-[:Contains]->(s:Synapse)

    // -- Filter synapse 's' --
    WITH n, s
    WHERE (s.`EB`) AND (s.type = 'post')

    // De-duplicate 's' because 'pre' synapses can appear in more than one SynapseSet
    WITH DISTINCT n, s

    RETURN n.bodyId as bodyId,
           s.type as type,
           s.confidence as confidence,
           s.location.x as x,
           s.location.y as y,
           s.location.z as z,
           apoc.map.removeKeys(s, ['location', 'confidence', 'type']) as syn_info